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by 

Eric  S.  Tollar 

ABSTRACT 

A  statistical  method  is  proposed  to  analyze  certain  designs  for  program 
effect  when  no  proper  control  group  is  present.  It  is  shown  that  under  certain 
partially  verifiable  assuiq)tions  the  data  can  be  analyzed  by  standard  analysis 
of  variance  techniques.  If  an  assuiq>tion  of  no  interaction  is  doubtful*  then 
a  new  statistic  is  proposed  which  is  shown  to  have  an  asyiiq>totic  t  distribution 
The  analysis  is  then  carried  out  on  some  research  data. 


1.  INTRODUCTION 


In  the  field  of  education,  the  experimaitex  often  encounter  a  special  probleic 
in  the  statistical  design  of  his  experiment;  the  lack  of  a  control  group  (for 
example,  see  Hersberger(i989) .  Often,  the  scientist  has  devised  a  new  educaticmal 
program  which  he  believes  will  augment  or  improve  on  the  techniques  presently  used 
to  teach  skills  of  some  sort  to  students.  To  test  whether  this  new  program  is  in 
fact  an  improvement,  the  scientist  must  approach  an  educational  institute  and 
obtain  approval  to  teach  his  program  to  the  target  students.  Since  most  of  these 
programs  appear  beneficial  to  the  students,  the  institute  usually  agrees  to  having 
the  program  implemented  with  one  stipulation;  that  everyone  who  qualifies  for  the 
program  is  admitted  into  it.  As  such,  the  experimenter  is  not  allowed  to  split 
the  students  into  an  experimental  groiq)  and  a  control  group,  and  the  control  grotq> 
must  be  forgone.  Often,  nothing  statistical  can  be  salvaged  from  the  data,  but 
we  suggest  a  statistic  to  measure  the  efficacy  of  the  program  for  certain  set-»q>s 
of  the  experiment  when  no  control  group  is  available. 

The  experimental  set-up  we  address  is  one  in  which  even  in  the  experimental 
school,  not  all  of  the  students  available  for  the  program  actually  qualify.  This 
situation  is  especially  common  in  the  field  of  gifted  or  special  education,  where 
only  those  students  who  achieve  some  minimum  (or  maximum)  score  on  an  entrance 
exam  are  admitted.  Further,  we  assume  there  are  other  schools  available  where  the 
program  will  not  be  implemented,  but  the  tests  to  determine  admittance  and  to 
measure  increases  in  learning  skills  can  be  administered.  These  schools  we  call 
the  comparison  schools. 

Typically  we  see  in  the  analysis  of  data  by  standard  techniques  that  these 
other  schools  are  pooled  together  and  used  as  a  control  group.  Often  there  is 
substantial  evidence  that  these  schools  cannot  be  considered  as  members  of  the 
same  population,  and  as  such  the  pooling  is  improper.  An  allowance  for  the 


"school"  effect  must  be  made.  We  deal  with  the  case  in  which  the  "school"  effect 
is  clearly  present^  and  the  typical  analysis  is  therefore  inappropriate. 

In  section  2  we  specify  the  statistical  model  used  to  describe  the  data, 
and  show  that  frequently  the  data  can  be  analyzed  properly  using  standard  analysis 
of  variance  tediniques.  In  sectitm  3  we  deal  with  the  case  where  the  assumptions 
for  the  standard  analysis  of  variance  technique  are  not  satisfied,  and  recomnend 
an  alternate  statistic.  It  is  shown  that  under  reasonable  assun^titms  this  dis¬ 
tribution  will  be  approximately  a  t-distribution.  In  section  4  we  analyze  the 
data  obtained  by  Hersberger  (1983), using  the  proposed  statistic. 


2.  THE  MODEL 


Let  represent  the  final  scores  of  the  students  (these  scores  we  assume 


to  be  the  only  nunbers  of  interest,  they  may  be  of  the  form  post-test  minus  pre¬ 
test  or  other  similar  transformation  of  the  data.)  The  value  of  i  indicates  the 
school  of  the  student,  so  the  range  of  i  will  be  0  s  i  s  la,  where  i  =  0  indicates 
a  student  from  the  program  school.  The  value  of  j  indicates  if  he  qualified  for 
the  program  (j=l),  or  did  not  qualify  for  the  program  (j=0),  regardless  of  school. 
The  value  of  k  merely  distinguishes  which  student  in  the  various  classes  we  have. 


where  1  s  k  s  n^ ^ , 


We  then  assume  for  Osisra,  Osjsl,  Isks  n^^^ 


‘ijk  '  “i  •  •  j  •  'ijk 


(2.1) 


where  N(0,o^),  i.i.d,  for  all  i,  j,  k,  and  6^^  'v  independent  of 


and  6.,  i*  j,  for  some  distribution  function  F, . 


Except  for  the  distribution  of  the  6^'s,  this  is  easily  seen  to  be  simply  a 
reparametricization  of  the  standard  2xin  analysis  of  variance  design.  The  6^'s 
measure  the  expected  differences  in  the  scores  of  the  qualifying  and  non-qualifying 
students  of  the  same  school. 

As  a  special  case  of  the  general  model,  we  consider  the  case  where  6^^  « 
»...»£  sj  for  some  constant  5.  In  this  case,  it  is  easy  to  see  that 
5g  -  6  measures  how  much  additional  average  difference  qualifying  students  have  over 
non-qualifying  students  when  compared  to  qualifying  students  for  the  conq>arison 
schools.  Unless  there  is  a  reason  for  believing  the  program  school  is  special 
(without  the  presence  of  the  program) ,  the  effect  6q  -  6  must  be  attributed  to  the 
program.  We  therefore  call  6q  -  6  the  program  effect. 

By  straightforward  analysis  of  variance  techniques  (see  Graybill  (1976)),  it 
can  be  seen  that  the  best  linear  unbiased  estimate  of  -  6^^  is 


where 


*ijk  * 


(2.5) 


and 

”i  “  "i0"il^"i0*"iP'^  ,  0  £  i  s  m.  (2.6) 

Further,  for 


where 


”ij 

X  ^*iik 
k=l 


then  it  can  be  easily  established  that 


o-«i>  ® 


(2.7) 


has  a  t-distribution  with  nonceittrality  paraaeter 


X  »  (6--«)  o’^Cw  "^  ♦  (  I  wp"‘]"*'^  . 
0  “  i.l  ^ 


and  degrees  of  freedom 


(  J  n.«  ♦  n..)  -  2(m*  1). 


Of  course,  this  test  statistic  can  be  implemented  by  most  statistical  coaputer 
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packages  by  simply  specifying  the  contrast  £/)  -*  m  i  6.  . 

^  i«l  ^ 

To  test  the  assumption  that  6j  =  fij  ®  •••  ®  ®aj  “  **  could  tenporarily 
delete  the  program  school  from  the  data  set  and  nm  an  analysis  of  variance  with 
a  high  significance  (say  a  »  .5  or  .75)  to  test  for  interaction.  If  there  is  no 
evidence  of  interaction,  it  seems  reasonable  to  accept  that  5^  *  ^2  *  •••  ® 

The  statistic  in  (2.7)  could  then  be  used  to  test  for  significance  of  the  program 
and  give  an  estimate  of  the  quantitative  average  effect  of  the  program. 

For  the  data  of  Hersberger(1983),  it  was  found  (perhaps  surprisingly)  that 
on  the  majority  of  measures  there  was  no  interaction  in  the  control  schools,  and 
as  such  there  is  reason  to  believe  the  simple  test  procedure  outlined  above  is 
appropriate  in  numerous  experiments  of  the  prescribed  form. 


3.  THE  STATISTIC  WHEN  INTERACTION  IS  PRESENT 


Of  course,  occasionally  there  will  be  evidence  of  interaction  at  an 
unacceptable  significance  level.  In  such  a  case,  the  simple  procedure  outlined 
above  is  not  applicable,  and  we  must  consider  the  more  general  model.  Even  in 
this  case,  we  consider  only  a  special  case  of  the  model  (2.1). 


.  .  ■ .  •  .■V . 


w  assume  that  5^,  ...»  5^  is  a  random  sample  from  some  normally  distributed 
population.  As  such,  we  have 


6^  '»•  N(«,n  )  ,  1  s  i  s  m. 


(3.1) 


We  assume  that  5^  is  from  a  normal  population  with  the  same  variance  and  arbitrary 


raean,  that  is 


Sq  N(Y,n 


(3.2) 


Of  interest  is  y-5:  that  is,  are  6.  and  (6,,  ...,  6  }  all  elements  of  the 

u  1  m 

same  population?  We  will  call  y  -  6  the  expected  program  effect,  since  it  is  the 
additional  difference  expected  in  the  program  school  scores. 

In  this  case,  we  propose  a  new  statistic,  given  by 

t*  =  (yo-«-)C(m-l)”^  X  W*(y  W*)"^]"^^^  (3.3) 

"  i,l  ^  ^  i=l  t 

where 


^i  =  ’^ii  -  *i0 


wj  =  (n2/;2  .  , 


(3.4) 


(3.5) 


a'=(I  i(n  -l)s2,/f  f(n  -1). 

i=0  j=0  i=0  j=0 


(3.6) 


«  IQ  ^  ^  If  ^ 

.-1  r  -^2  ^2  -i  r 


=  [(m-l)"^  I  iy.-y)^  -  o^m"*  I  (n-g"^*  n.  "^)]*, 

i»l  ^  i=.l 


*  I  **1*//  ?  w  *  . 
i»l  i«l  ^ 


(3.7) 


(3.8) 


The  main  theorem  of  this  paper,  dealing  with  the  asymptotic  distribution  of 


t*.  we  now  state. 


THEOREM  3,1,  Under  the  assiaptions  (2,1),  (3.1),  (3,2),  as 

min  (n. .)  •*•  «,  P(t*  s  x) -*■  P(tsx), 

Osism 
Osjsi 

where  t  has  a  t~distribution  with  m  -  1  degrees  of  freedom  and  non-centrality 
parameter  X  »  (y  -  6)/n/l  ♦  m"^  . 

Before  proving  the  theorem  we  establish  some  intermediate  results,  which  in 
addition  to  the  proof  of  the  theorem  will  shed  li^t  on  the  motivation  for  the 
statistic  t*.  The  first  result  is  stated  without  proof  as  it  is  an  elementary 
result  which  can  be  easily  established  using  standard  techniques  (see  Graybill 
(1976)). 

LEMMA  3,2,  If.  Z ^ ,  . . . ,  are  independent,  normally  distributed  random  variables 
with  mean  6  and  variance  respectively,  then 

(1)  lyie  BLUE  for  6  is  6  =  (Iwi)"^(Iw^Zj) ,  ^ 

(2)  c-2^w.(z. -6)^  =  a'^Clw.^z.  .( Jw.)‘^Iw.zp^]  has  a  x^(ia-l)  distribution, 

A 

independent  of  6  , 

Thus,  to  understand  the  motivation  of  the  statistic  t*,  consider  the  quan¬ 
tities  y^  =  It  is  clear  that 

y^  -v  N(6,  Ti^  +  o^(nj^j"^-^  n^Q^  )),  1  s  i  s  m,  and  (3. 


Yq  N(y,  n^  +  a^(n^j‘^ 


(3. 


2  -2 


If  n  o  were  known,  then  letting 


from  the  previous  leimna  the  statistic 

(3.12)  (yo-S)/C(  I  w.(y. J  w  )’^)] 

i=l  ^  ^  i=l  ^ 

would  then  have  a  t-distributicm  with  m-  1  degrees  of  freedom.  The  idea  of  t*  is 
2  2 

to  estimate  q  /o  and  use  these  to  provide  an  estimate  of  the  w^'s.  We  know  that 

2  ^2 

a  has  a  very  reliable  estimator  available,  o^  given  in  equation  (3.6). 

-1  “ 

It  can  be  quickly  determined  that  for  y  =  m"  I  y- , 

i-1  ^ 

“ ill'*’!  ■  ^ 

Therefore,  we  can  use  as  an  estimate  of  n  the  statistic  given  in  equation 
(3.7),  and  using  w^  in  equation  (3.5)  as  an  estimate  of  w^  in  equate  m  (3.22)  w« 
find  the  resultant  statistic  is  t*. 

To  establish  that  t*  is  indeed  asymptotically  t-distributed,  we  need  the 
following  lemma. 

LEMMA  3.3.  If  {X^:  n  =  1,  2,  ..  J  is  a  sequence  of  random  variables  wher*;^ 


1)  for  some  measure  p,  has  density  f^  with  respect  to  p 

2)  f ^  s  k  •  g  for  all  n,  and  some  density  g,  and 

3)  f^  -*■  f ,  for  f  a  density, 

then  for  any  function  h(x,y)  where  h(x,y)  is  continuous  with  respect  to  y  for 
U-almost  all  x,  and  for  any  function  Six)  where  ^  3^, 

hCX^,  ^(Xj^))  ^  h(X,6Q),  where  X  has  density  f. 

Remark:  Condition  3)  can  be  weakened  to  X  ^  X,  some  random  variable,  and  the 

result  is  still  true.  It  should  be  pointed  out  that  conditions  1)  and  2)  coupled 
with  X^  ^  X  does  not  imply  f^  -*■  f .  However,  this  weakening  only  complicates  the 
proof  of  the  lemma,  and  since  theorem  3.1  only  requires  the  lemma  as  stated,  we 


do  not  prove  the  lemna  under  the  weaker  condition 


PROOF  OF  LEMMA  3.5.  By  the  dominated  convergence  theorem  (see  Chung  (1974))  we 
have  that 

lim  /  e^*’'^***o)f^(x)dM(x)  =  /e^^’'^^*^0^f(x)du(x) , 
n  -*■<» 

d 

and  as  such,  h(X^,6Q)  h(X,6Q).  Therefore,  to  complete  the  proof,  we  need  only 

show  that  h(Xj^,6(X^))  -  h(X^,6Q)  5  0.  For  y  >  0,  let 

A(y)  =  {x:  ||h(x,6)*  h(x,6Q)|l>e,  some  6  where  ||  6  -  6q  |1  ^  yJ  •  (3.14) 

Since  h(x,6)  is  continuous  with  respect  to  6,  as  y +  0,  A(y)  +  A,  where  w(A)  =0. 

Let  Y  be  a  random  variable  with  density  g  of  condition  2) .  Then 

lim  P(Y€A(y))  =0.  As  such,  for  all  e  >  0,  there  is  a  y- >  0  where  for  all 
y-K)  " 

y<yQ»  P(YeA(y))  <  e/k.  Thus,  for  all  n,  if  y  <  y^  then 

P(Xn€A(y))  sk  P(Y£A(y))  <  e.  (3.15) 

Choose  an  N  where  for  n  >  N,  y  <  y^,  P(|l  4(Xjj)  -  5q  |1  >  y)  <  e.  Then  we  have  that 

P(l|h(Xjj,  6(X^))  -  h(X^,  6q)1|>  e)  ^  P(X„  €  A(|l  6  (X^)  -  6^  1|))  .  (3.16) 

Observing  that 

P(X^eA(||  6(X^)-«q11))  SP(X^£A(|1  6(X^)  -  |1)  ,  ll6(Xj^)  -  «q||s  y)  ♦  e  (3.17) 

S  P(X^  €  A(y))+  e  <  2e, 

p 

we  have  that  h(X^,  6(X^))  -  h(X^,«Q)  0  ,  which  completes  the  proof  of  the 

lemma.  □ 

We  now  proceed  to  establish  theorem  3.1  as  a  consequence  of  lemma  3.3. 


PROOF  OF  THEOREM  3.1 


Let  us  first  establish  that  for  as  in  (3.5),  we  have 


(  I  Wj*  )  ^  CV'  ”1*’  *  •  *  ’  ^  ^  m’  * '  •  ’  m  ^  * 

1=1 


(3.18) 


2  2  1  1 

First  observe  that  for  independent  y.  where  y4  'v-  N(y,  n  *0  (- —  ♦  - —  )),  and 

1  1  "il  ”io 

for  the  density  of  g(x)  of  y'''N(Y,  +  g(x)/l  +  2  aVn^  clearly  dominates  the 

density  of  each  y. .  Also,  as  min  (n.  .)  each  density  clearly  converges  to 

^  0Si<m 

0<jsi 

that  of  N(y,  n^)«  Thus,  by  the  previous  lemma  (or  more  simply  by  the  Skorohod 

in  2  d  2\f  2 

convergence  theorem,  see  Serfling  (1980),  we  have  that  J  (y.  -  y)  -►  (m-1). 

i=l  ^ 


^2  P  2 

Also,  since  we  have  that  a  -*■  a  ,  we  get  from  Slutsky's  theorem  (see  Chung  (1974)) 
2  2  ^  ^  ^  2 

that  n  /a  -*■  —27 — X"^(m-1).  We  can  therefore  find  an  N  where 
c^im-l)  1 

P(n^/o^  s  6)  <  e  for  min  (n..)>N.  , 

Osism 


Osjsl 


Letting  c^^ 


+  n 


we  can  establish  that 


C  max  (c.)]”^-c."^ 
isism  _ _ 

m(n^/a2)  +  c^^ 


i=l 


I 

m 


[  min  (c.)]"^-c.”^ 
^  lsi<m  _ _ 

m((^i2/52)  +  c.-l 


(3.18) 


for  0  s  i  s  m  . 
Therefore 


* 


[  min  (c.)3”^  C  max  (c. )] 


so  we  have  that  for  min  (n. .)  >  N. , 

Osism 

Osjsi 


w. 
>  m 


*  min  (c.)"^-  max  (c.)”^ 

1^  I  ^  l^i^m  l^i^m 

?  .  ’  “  m6 

I 

i=l 


)  >  1  -  e  . 


From  (3.20),  it  is  clear  that  ( 


w  * 

,  "o 

W  * 

''i 

W* 

m 

'  ffl 

*  m  * 

•  •  •  > 

m 

l-i 

.S"i* 

1=1 

1=1 

m 

i=l 

(fio*** 

=  1 
i=l 

Vi* 

(3.20) 


h(x,6)  = 


Xo-x 


is  clearly  continuous  for  all  fixed  x  with  respect  to  6.  Letting  =  (y^j, 


y  w.* 
i=i  ^ 


)  ,  we  have  that  h(Y„,  6(X„))  4  h(X.(i,i, 

i=l  ^ 


2  2 

where  '''  n(5»  n  ),  is  1,  X^  -v  N(Y,n  ),  and  X^’s  are  independent. 

Clearly  h(X,(— ,  ...,^))  t(m-l),  which  complete  the  proof.  □ 

*^111  in 


In  the  following  section,  we  examine  the  data  of  Hersberger  (1983),  to  show 
how  the  techniques  in  this  and  the  preceding  section  can  be  used  to  analyze  the 
effects  of  a  program  administered  under  the  prescribed  set-up. 

4.  ANALYSIS  OF  DATA 

In  19S3,  Hersberger  administered  a  program  to  measure  the  effectiveness  of 
computers  in  the  teaching  of  mathematics  to  students  of  high  and  low  mathematical 


sir- 


ability  (as  defined  by  those  students  who  scored  in  the  upper  1/3  and  lower  2/3 
on  a  math  pre-test,  respectively).  While  we  certainly  do  not  analyze  all  the 
data  of  the  study,  we  do  select  some  particular  measurements  to  illustrate  the 
application  of  the  analysis  outlined  above.  The  two  measures  selected  are  a 
computational  measxxre,  and  a  value  of  mathematics  measure. 

Below  is  a  listing  of  the  class  means,  standard  deviations,  and  sample  sizes 
for  the  computational  measure,  with  a  corresponding  plotting  of  the  class  means. 

Table  1.  Summary  of  scores  on  the  computational  measure 


School 


Mathematical 

Ability 


Program 

School 


Comparison 
School  1 


Comparison 
School  2 


Comparison 
School  3 


x=  34,67 

X  a  29.96 

X a  24.00 

X  a  28.43 

s  =  5,19 

s  ®  4*36 

s  a  4.81 

S  a  2.76 

n  a  15 

n  a  9 

n  a  10 

n  a  7 

x=  18.97 

xa  24.10 

X  a  17.76 

X  a  22.31 

s  a  5.66 

s  a  5.55 

s  a  5.18 

S  a  6,05 

n  a  37 

n  a  20 

n  a  17 

n  a  13 
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Figure  1.  Cell  means  for  the  computational  measure. 

A  test  for  interaction  with  the  program  school  deleted  yields  on  F-value 
of  .04,  which  has  a  level  of  significance  of  .96.  As  such,  the  procedure  out¬ 
lined  in  section  2  using  a  conventional  analysis  is  appropriate.  From  equations 
(2.4)  and  (2.7)  we  find  that 

«  9.7759  , 

3^  *  28.2516  ,  and 

t  =  4.7123  with  df=120. 

Thus,  we  find  that  the  program  effect  is  significant  at  the  p  =  .000006  level ^ 
and  conclude  the  program  has  a  significant  effect  on  computational  skills. 

The  next  measure  we  consider  is  the  value  of  mathematics  measure.  Below 


is  a  listing  of  the  class  means,  standard  deviations  and  sample  sizes,  with  a 
plotting  of  the  class  means. 


1.51,  i^ich  has  a  .23  level  of  significance.  As  such,  the  assumption  of  equality 
of  the  differences  may  be  suspect,  so  the  analysis  outlined  in  section  3  is 


applied.  Application  of  equations  (3.3)  through  (3.8)  yield 
«  25.8789,  „  ^gi92  ^ 

Wq*  »  8.9232,  w*  *  5.9736,  v*  •  5.3641,  w*  =  4.1033, 

6'  «  1.6182  and  t*  =  ,7963. 

The  approximate  level  of  significance  of  the  statistic  t*  with  2  degrees  of 
freedom  is  p  »  .2547  .  That  the  level  of  significance  is  relatively  high  should 
be  no  surprise,  for  most  of  the  significance  can  be  attributed  to  the  negative 
difference  of  coiiq)arison  school  1. 

5.  CONCLUSION 

In  the  analysis  outlined  in  sections  2  and  3,  deleting  the  program  school 
then  testing  for  interaction  between  school  and  ability  at  a  high  significance 
level  is  crucial.  The  entire  analysis  of  section  2  hinges  on  the  assumption  that 
the  average  difference  between  high  and  low  ability  students  in  the  abscence  of 
any  program  is  constant  across  schools,  and  without  that  assumption,  none  of  the 
results  are  justifiable. 

The  assumptions  in  section  3  that  seem  the  most  suspect  are  the  assumption 
that  the  6j^'s  have  a  normal  distribution,  and  the  assumption  that  6q  has  the 
same  variance  as  the  others.  Clearly,  some  assumption  on  the  distribution  of  the 
5^*s  must  be  made,  since  m  is  typically  small.  That  the  distribution  should  be 
normal  seems  as  reasonable  as  any,  and  makes  the  mathematics  easy.  The  variance 
of  is  unidentifiable,  so  the  assumption  of  equality  of  variance  seemed  neces¬ 
sary. 

Of  course,  even  if  the  assumptions  in  section  2  or  section  3  seem  justifiable, 
the  analysis  above  can  never  be  considered  as  a  replacement  for  a  good  experi¬ 
mental  design  with  a  control  group.  The  assumptions  necessary  seem  critical. 


and  minor  deviations  may  cast  doubt  upon  the  conclusions.  Ihis  analysis  is 
suggested  only  when  a  control  groi^)  is  definitely  unattainable. 

Lemna  3.3  is  clearly  applicable  to  distributions  on  the  6.'s  other  than  the 
normal  distribution,  and  as  such  an  examination  of  other  distributions  may  prove 
interesting.  However,  such  alternate  distributions  usually  are  not  employed  in 
most  designs  employed  by  applied  statisticians,  so  we  leave  this  as  an  area  of 
further  research. 

Of  major  interest  is  that  under  the  same  assumptions  made  in  section  3, the 


C(m-l)"^  I  y?  -m‘^(  I  [m“^m+l)]^^2 

i=l  ^  i»l  ^ 


also  has  an  asymptotic  t  distribution  with  m  -  1  degrees  of  freedom,  and  that  in 
fact  this  can  be  more  easily  established  by  a  straightforward  appeal  to  the 
Skorohod  convergence  theorem  (see  Serf ling  (1980)).  Therefore,  the  major 
jiistification  of  the  use  of  the  statistic  in  section  3  is  the  conjecture  that  it 
should  outperform  t  for  small  n^^'s.  This  conjecture  is  an  open  question  which 
could  perhaps  be  answered  by  computer  simulations  or  some  other  technique. 
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